Identifying 
Educational 
Practices 
Supported by 
Rigorous Evidence 


A Guide to the Selection 
of Evidence-Based 

Practices 






RRCP. 

Regional Resource X 
Center Program 


A Publication of the 

Spring 2014 

Regional Resource Center Program 


Inside 

Six Steps to Guide the 
Process of Selecting 
Evidence Based 
Practices (EBP): 

(1) Begin with the End in 
Mind - Determine 
Targeted Outcomes 

(2) Identify Potential 
EBP(s). 

(3) Determine Evidence 
of Effectiveness of 
Potential EBP(s). 

(4) Analyze 
Organizational Fit of 

Potential EBP(s) 

(5) Analyze 

Organizational Capacity 
and Resources Relative to 
the EBP(s) 

(6) Consider Monitoring 
and Sustainability 


An Understanding of Evidence-Based Practices 

One component of the recently required State Systemic Improvement 
Plan (SSIP) for State Departments of Education calls for the selection and 
implementation of evidence-based practices (EBPs). Specifically, 
language within Indicator 17 of the 2015 Part B SPP/APR and Indicator 
11 of the Part C SPP/APR requires "LEA/EIS implementation of EBPs to 
improve the State-identified result(s)" for infants, toddlers, children and 
youth with disabilities. 

McColskey & Lewis (2007) note that EBP means delivering services to 
students (clients) in ways that integrate (a) best available evidence from 
data, research, and evaluation; (b) professional wisdom gained from 
experience; and (c) contextual knowledge of the particular classroom, 
school, district, or state that might impact the design or implementation. 
More recently, Dean Fixsen and colleagues at the University of North 
Carolina at Chapel Hill (Fixsen, Blase, Metz & Van Dyke, 2013) proposed 
a formula for successful uses of evidence-based programs in typical 
human service settings that can be characterized as: 

Effective Interventions x Effective Implementation = Improved Outcomes 

Note that the formula for success involves multiplication. If 
interventions are not effective (intervention = zero), then the intended 
outcomes will not be achieved. If implementation supports are not 
effective (implementation = zero), then the intended outcomes will not 
be achieved. This document is meant to assist state and school leaders 
in making decisions around this formula - particularly decisions 
around selecting EBPs that have the greatest potential to result in 
improved outcomes for all students. 
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The U. S. Department of Health and Human Services' document Iden tijying and 
Selecting Evidence-Based Programs and Practices: Questions to Consider provides 
helpful steps and key questions to consider when charged with selecting EBPs 
that meet the needs of a target population. Using this document as a framework, 
the following steps, along with suggested tools and resources, provide a 
recommended route for identifying and selecting educational EBPs that will 
closely align with the needs of your state, district or school. 



( 1 ) Begin with the End in Mind - Determine Targeted Outcomes 

Begin the process by determining what it is you would like your target population to be able to know or 
do as a result of taking part in the EBP or program. The following questions can help in identifying 
outcomes for your targeted population in specific, observable and measurable terms: 

Outcome Identification 

© Who is the target population? 

® What is the issue you are trying to address? 

® Why or how did the issue develop? 

® What variables contribute to the issue? 

® What is currently being done to address the issue? Why is this not sufficient? 

® What are the desired outcomes from School, District, and/or State Level perspective^ )? 

Conducting a thorough data analysis and developing a detailed Theory of Action or Logic Model can help 
generate answers to these questions. Logic models help plan with the end in mind - the targeted 
outcomes. Their purpose is also to communicate the underlying "theory" or set of assumptions or 
hypotheses that program proponents have about why a potential program will work, or why it is a good 
solution to an identified problem. For more information about logic models, see your RRCP liaison. 

(2) Identify Potential Evidence-Based Practices (EBP) 

Once outcomes have been clearly established, the next step is to identify potential EBPs that target the 
variable(s] contributing to the cause of the issue(s] you are trying to address. Before beginning this 
process, consider the following questions: 

EBP Identification 

© What are the clearly defined, specific skills or knowledge sets being targeted? 

© What types of students will the EBP target (e.g., all students, Tier 3 students, EL special education 
students, Grades K-5 students, students with Autism, etc.)? 

© What is the intended reach and scope of the EBP (e.g., individual skill change, population-level skill 
change, etc.)? 

There are several different types of sources to help identify interventions appropriate to targeted 
needs. Although not inclusive of all potential sites, the following resources can aid in this search. 

Related National and Federal Websites 
What Works Clearinghouse 
http://ies.ed.gov/ncee/wwc/ 

The What Works Clearinghouse (WWC) was established in 2002 as an initiative of the Institute for Education 
Sciences (IES) at the U.S. Department of Education. The WWC is administered by the National Center for 
Education Evaluation within the IES. The goal of the WWC is to be a resource for informed educational decision- 
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making. To reach this goal, the WWC identifies studies that provide credible and reliable evidence of the 
effectiveness of a given practice, program, or policy (referred to as "interventions") and disseminates summary 
information and reports on the WWC website. 

National Secondary Transition Technical Assistance Center (NSTTAC) Evidence-Based Practices 
http://www.nsttac.org/content/evidence-based-practices 

NSTTAC conducts reviews of high quality group experimental (Gersten et al., 2005) and single-subject research 
(Horner et al., 2005) to identify evidence-based practices in the area of secondary transition services (Innes- 
Helsel, et al., 2006; Twyman & Sota, 2008). Practice descriptions organized by intervention (e.g., community-based 
instruction, response prompting) and by skill taught (e.g., academic skills for students with disabilities at the 
secondary level, completing a job application, grocery shopping) are available, as well as corresponding Lesson 
Plan Starters. 

The IRIS Center Online Tools 

http://iris.peabody.vanderbilt.edu/ebp/ 

The IRIS Center at Vanderbilt Peabody College has developed an online tool of Evidence-Based Practice Summaries. 
These summaries of research about the effectiveness of instructional strategies and interventions contain links to 
research reports and include information about an intervention’s level of effectiveness and the age groups for which 
it is designed. 

State Implementation & Scaling-up of Evidence-based Practices Center 

http:/ / sisep.fpg.unc.edu/ 

The purpose of the SISEP Center is to help to establish implementation and scaling capacity in state, regional, and 
district educational systems. SISEP provides content and technical assistance toward establishing large-scale, 
sustainable, high-fidelity implementation of effective education practices. The shared goal is to maximize 
academic and social outcomes of all students, especially those students with disabilities. Among its many services, 
the Center offers tools and resources for conducting this work, including formative and summative evaluation 
tools for action planning, monitoring and outcome assessment. 

Additional Sites 

National Autism Center (NAC) 

http:/ /www.nationalautismcenter.org/ 

The National Autism Center is dedicated to serving children and adolescents with Autism Spectrum Disorders 
(ASD). NAC has developed guidelines and standards for identification of evidence-based practices to help 
researchers and practitioners make informed choices regarding interventions for students with autism. NAC has 
completed the National Standards Project that provides a set of standards for effective, research-validated 
educational and behavioral interventions for children/adolescents with autism. NAC has introduced an Educator 
Manual - Evidence-Based Practices and Autism in Schools, which is available for download on the NAC website. 

National Professional Development Center on Autism Spectrum Disorders 

http:/ / fpg.unc.edu / news / research-backs-more-strategies-children-autism 

The National Professional Development Center on Autism Spectrum Disorders Evidence-Based Practices for 
Children , Youth, and Young Adults with Autism Spectrum Disorder is another helpful document. Scientists at UNC's 
Frank Porter Graham Child Development Institute led the project, screening 29,000 articles about autism spectrum 
disorders (ASD) to locate the soundest research on interventions for children from birth to age 22. The purpose of 
this report is to describe a process for the identification of evidence-based practices (EBPs) and also to delineate 
practices that have sufficient empirical support to be termed "evidence-based." 

Promising Practices Network (PPN) 
http://www.promisingpractices.net 

The RAND Corporation operates this project. RAND is a nonprofit research organization providing objective 
analysis and effective solutions that address challenges facing the public and private sectors around the world 
objectivity. The wide dissemination of its findings is an equally important part of RAND's mission of making 
research readily accessible to policymakers and others if it is to have impact on the public good. 
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In addition to reviewing the research on 
different programs , products, practices, 
and policies in education, the What 
Works Clearinghouse has produced a 
series of Practice Guides that 
summarize evidence-based practices 
and provide teaching recommendations 
in key academic areas. These can be 
directly downloaded from the WWC 
website. Among these Practice Guides 
are the following publications: 

• Teaching Math to Young Children 

• Assisting Students Struggling with 
Mathematics 

• Improving Mathematical Problem 
Solving in Grades 4-8 

• Improving Reading Comprehension in 
Kindergarten Through 3 rd Grade 

• Assisting Students Struggling With 
Reading 

• Teaching Elementary Students to 
Become Effective Writers 

• Dropout Prevention 

• Helping Students Navigate the Path to 
College: What High Schools Can Do 

• Improving Adolescent Literacy: 
Effective Classroom and Intervention 
Practices 


http://ies.ed.gov/ncee/wwc/Publications R 
eviews.aspx?f=All%20Publication%20and% 
20Review%20Types.3:#pubsearch 


Technical Assistance and Dissemination Network 
(TA&D Network) 

The TA&D Network is a network of approximately 45 Centers, 
including the Regional Resource Center Programs, funded by the 
Office of Special Education Programs (OSEP) and provides 
information and technical assistance to states, schools, educational 
professionals and families on topics related to special education. A list 
of these centers can be found at http://www.tadnet.org/pages/526- 
find-a-center . While all of the centers offer important information, we 
highlight three of these centers that may be of particular help with 
identifying evidence-based practices for students with disabilities. 

National Center on Intensive Intervention NAHONAl CENTER ON 

INTENSIVE INTERVENTION 

http://www.intensiveintervention.org/ 

The Center is designed to help schools work with students with severe 
and persistent learning or behavioral needs. Within multi-tiered 
systems of supports, this is often referred to as Tier III instruction. 
Among its resources, the Center publishes a set of tools charts focused 
on academics and behavior to assist educators and families in 
becoming informed consumers about intensive interventions. The 
Center also provides tool charts that assist practitioners in selecting 
intervention platforms and progress monitoring tools that best meet 
the needs of their students. 

National Dropout Prevention Center for Students with 
Disabilities 

http://www.ndpc-sd.org/ 

The overall goal of this Center is to provide high-quality, evidence- 
based technical assistance to help states build and implement 
sustainable programs and best practices that will yield positive results 
in dropout prevention, reentry, and school completion for students 
with disabilities. Relatedly, the National Post-School Outcomes 
Center (psocenter.org) has recently produced several products 
related to post-school success including A ligning Evidence-Based 
Practices and Predictors for Post-School Success that highlights 
evidence-based practices that support implementation of in-school 
predictors of post-school success. 

«cTd» 

The Family Center on Technology and Disability 
www.fctd.info /resources 

The Family Center on Technology and Disability (FCTD) is a resource 
designed to support organizations and programs that work with 
families of children and youth with disabilities. The Center offers a 
range of information and services on the subject of assistive and 
instructional technologies. Among these is the FCTD resource review 
database. Hundreds of assistive and instructional technology 
resources have been identified, reviewed, and annotated by topic, 
disability, and material type. Direct links to each resource have been 
provided in most cases as well. 
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National Center on Accessible Instructional Materials and National Center on Universal Design 
for Learning 

http://aim.cast.org/ and www.udlcenter.org 

Many educators are aware of the Center for Applied Special Technology (CAST), but have not taken the 
opportunity to become familiar with the plethora of strategies, tools, research reports and other available 
materials for practitioners to assist students with disabilities gain access to the core curriculum. Both of these 
websites, along with the CAST website at www.cast.org. may be quite helpful in providing solutions to everyday 
challenges for students with varying disabilities. 

Peer-Reviewed Journals 

Peer-reviewed journals are another important source for identifying evidence-based practices and 
programs, including those not typically listed on prominent review sites. When reviewing articles on a 
particular intervention, it is wise to look at the entire body of research (all relevant documents) on an 
intervention rather than draw conclusions from a single piece of research. This helps ensure that the 
conclusions that are drawn are based upon a consistent pattern of results rather than results from one 
single study. 

For practitioners, using this type of literature is often very time consuming and resource intensive, 
particularly without ready access to academic libraries and electronic copies of journal articles. Also, it 
oftentimes requires technical expertise to review the literature and interpret results. Studies vary 
greatly when it comes to the quality of the theory on which the intervention is based, experimental 
design, measurement strategies used to evaluate the results and conclusions presented. In the next 
section, please find several key elements and questions to consider when analyzing articles in peer- 
reviewed journals. 

(3) Determine Evidence of Effectiveness 

After identifying a potential EBP that targets the variable(s) of concern, it is important to consider the 
strength of evidence behind the potential intervention. There are many types of designs that can be 
used in research studies. For example, the most rigorous studies of program effectiveness will use a 
research design called random assignment, which compares an intervention's impact on students or 
teachers to that of a control group that didn’t get the intervention. Such experimental studies (with 
random assignment to the treatment and control groups) are costly and difficult to do, and often take 
several years to complete. Thus, not many of the programs or strategies considered will have this level 
of evidence. Another research design that is much more common is quasi-experimental design. This 
design also compares an intervention’s impact on students or teachers to that of a control group that did 
not get the intervention. However, the participants were not necessarily randomly assigned to the 
groups. Well-done quasi-experimental studies that have been replicated also provide good evidence of 
effectiveness. Single-subject studies can also be informative. Single-subject experimental designs use 
participants (individuals or groups) as their own control and collect repeated measures of dependent 
variables over time to test the effects of a practice manipulated by the researcher. 

For a more thorough and in-depth understanding of the quality indicators in studies of practices in 
special education, the Council for Exceptional Children has recently commissioned researchers to 
develop, vet, and pilot new standards for determining evidence-based practices (EBPs) in special 
education. These standards are available through the CEC’s Standards for Evidence-Based Practices in 
Special Education available at 

http://www.cec.sped.org/Standards/Evidence-Based-Practice-Resources-0riginal7sc lang=en . 
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It is recommended that if using peer-review journals to determine the effectiveness of an intervention, 
individuals with expertise in this area be consulted. The following more general questions can also help 
guide discussions around evidence for effectiveness: 

Evidence of Effectiveness 

© What outcomes has the EBP achieved? 

© Has the EBP been implemented successfully within a similar setting and population as ours ( e.g . age, 
ethnicity, disability)? 

© Has the EBP been evaluated with a research design including a comparison or control group, with 
significant findings (simple pre-post studies and comparison-group studies in which the intervention 
and comparison groups are not well-matched do not provide solid evidence)? 

® Have the results been replicated in other studies? 

© Have evaluation studies found the EBP to be effective compared to other evidence-based practices? 

[4] Analyze Organizational Fit 

At this point, potential evidence-based practice(s) that have good evidence of effectiveness have been 
identified. The next step is to query whether the EBP is feasible for your organization taking into 
account organizational policies, procedures and structures. 

Organizational Fit Questions 

© Is the EPB culturally feasible, taking into account the mission, vision, values, and culture of the 
educational community? 

© Is the EBP compatible with priorities and current initiatives already in place at the State, District 
and/or School Levels? 

© Are there individuals that are able to "champion" the EBP? 

© Will and what leadership personnel will support the EBP implementation and sustainability? 

© What type of changes to the organizational framework will be needed in order to provide adequate 
support to frontline staff implementing the EBP? 

The National Implementation Research Network (NIRN) also has excellent resources that can help 
organizations work through needed discussions around selecting evidence based practices, particularly 
organizational fit and the organization's capacity and resources to support potential EBPs. In particular, 
NIRN’s Hexagon Tool can be used as a planning tool to evaluate evidence-based programs and 
practices during the Exploration Stage of Implementation. Information about the Hexagon Tool can be 
found at the NIRN website, http://implementation.fpg.unc.edu/resources/hexagon-tool-exploring- 
context. 

(5) Analyze Capacity and Resources 

An organization considering adoption of a particular EBP also needs to determine if it has the capacity 
and resources necessary to carry out the program or practice as it was intended. Essentially, the 
organization is looking to see if there is a match with the community's resources and capacity to 
implement a program — with the requirements of the program itself. For example, if considering 
implementation of a new intervention at a district or school level, many skills that are needed by 
practitioners can be initially introduced during formal training. However, research indicates these skills 
are best learned with follow-up through "on-the-job training" typically delivered by someone in an 
instructional coach position. As researchers Joyce and Showers (2002) indicate, training and coaching 
should be one 
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continuous set of operations designed to produce actual changes in the classroom actions of teachers. 
One without the other is insufficient. If the district or school does not have the capacity to support such 
needed coaching, it may not be the most feasible EBP for the district or school to select. 

t^mpg^ S t?ie r f^fo^Vng of questions can be helpful to this, process: . . , 

® Is tire EBPfrnanaally feasime, raking into account costs rtf purchase implementation materials and 

provide technical assistance? 

® How much will it cost to recruit, train, and fund additional staff to ensure staff competencies and skills 
match the EBP requirements (e.g., trainers, coaches, and implementation teams)? 

® Does the EBP fit in with scheduling demands of the State, District and/or School? 

© What are the EBP’s logistical requirements (e.g. technology, space, equipment, etc.)? 

© Do the benefits outweigh the costs of implementing the new EBP? 


(6) Consider Monitoring and Sustainability Issues 

Considerations for monitoring and sustainability of EBPs are also critical and should be discussed prior 
to adopting a potential EBP. Planning for sustainability should begin before implementation occurs. 
Additionally, a plan for evaluation of your desired outcomes should be developed prior to 
implementation so that effects of the practice or program can be determined. The following questions 
may be of assistance in working through these considerations. 

Monitoring and Sustainability Questions 

© What monitoring or evaluation tools are available to help track implementation quality (i.e., fidelity 
measures)? 

® Have the developers of the EBP measured the rate of success in sustaining their model with fidelity? 

© Do we have the resources and staff to ensure such tracking? 

© What are the long-term costs associated with maintaining the EBP and the organizational 
infrastructure required for sustainability? 

© What funding resources are available for continued support of the EBP? 

® Can we build mechanisms into the implementation process to ensure sustainability? 

In Conclusion 

The selection of EBPs that target desired outcomes is a challenging endeavor. Starting the process with 
clearly defining outcomes your state, district or school hopes to achieve is a critical first step and 
perhaps key to the entire process. Identifying potential EBPs and studying the evidence behind these 
potential interventions are important next steps, but clearly not enough. Spending thorough time and 
effort analyzing the organizational fit with potential EBPs, including organizational resources and 
capacity, is essential. 

By following the steps outlined in this brief, we hope you will be able to select and utilize the EBP(s) that 
most appropriately match your desired outcomes. For assistance and more information, please contact 
your Regional Resource Center Program. 


(Continued on next page) 
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